Rationality: Appreciating Cognitive Algorithms

Eliezer Yudkowsky6 Oct 2012 9:59 UTC
97 points
135 comments5 min readLW linkArchive

Followup to: The Useful Idea of Truth

It is an error mode, and indeed an annoyance mode, to go about preaching the importance of the “Truth”, especially if the Truth is supposed to be something incredibly lofty instead of some boring, mundane truth about gravity or rainbows or what your coworker said about your manager.

Thus it is a worthwhile exercise to practice deflating the word ‘true’ out of any sentence in which it appears. (Note that this is a special case of rationalist taboo.) For example, instead of saying, “I believe that the sky is blue, and that’s true!” you can just say, “The sky is blue”, which conveys essentially the same information about what color you think the sky is. Or if it feels different to say “I believe the Democrats will win the election!” than to say, “The Democrats will win the election”, this is an important warning of belief-alief divergence.

Try it with these:

  • I believe Jess just wants to win arguments.

  • It’s true that you weren’t paying attention.

  • I believe I will get better.

  • In reality, teachers care a lot about students.

If ‘truth’ is defined by an infinite family of sentences like ‘The sentence “the sky is blue” is true if and only if the sky is blue’, then why would we ever need to talk about ‘truth’ at all?

Well, you can’t deflate ‘truth’ out of the sentence “True beliefs are more likely to make successful experimental predictions” because it states a property of map-territory correspondences in general. You could say ‘accurate maps’ instead of ‘true beliefs’, but you would still be invoking the same concept.

It’s only because most sentences containing the word ‘true’ are not talking about map-territory correspondences in general, that most such sentences can be deflated.

Now consider—when are you forced to use the word ‘rational’?

As with the word ‘true’, there are very few sentences that truly need to contain the word ‘rational’ in them. Consider the following deflations, all of which convey essentially the same information about your own opinions:

  • “It’s rational to believe the sky is blue”
    → “I think the sky is blue”
    → “The sky is blue”

  • “Rational Dieting: Why To Choose Paleo”
    → “Why you should think the paleo diet has the best consequences for health”
    → “I like the paleo diet”

Generally, when people bless something as ‘rational’, you could directly substitute the word ‘optimal’ with no loss of content—or in some cases the phrases ‘true’ or ‘believed-by-me’, if we’re talking about a belief rather than a strategy.

Try it with these:

  • “It’s rational to teach your children calculus.”

  • “I think this is the most rational book ever.”

  • “It’s rational to believe in gravity.”

Meditation: Under what rare circumstances can you not deflate the word ‘rational’ out of a sentence?

...
...
...

Reply: We need the word ‘rational’ in order to talk about cognitive algorithms or mental processes with the property “systematically increases map-territory correspondence” (epistemic rationality) or “systematically finds a better path to goals” (instrumental rationality).

E.g.:

“It’s (epistemically) rational to believe more in hypotheses that make successful experimental predictions.”

or

“Chasing sunk costs is (instrumentally) irrational.”

You can’t deflate the concept of rationality out of the intended meaning of those sentences. You could find some way to rephrase it without the word ‘rational’; but then you’d have to use other words describing the same concept, e.g:

“If you believe more in hypotheses that make successful predictions, your map will better correspond to reality over time.”

or

“If you chase sunk costs, you won’t achieve your goals as well as you could otherwise.”

The word ‘rational’ is properly used to talk about cognitive algorithms which systematically promote map-territory correspondences or goal achievement.

Similarly, a rationalist isn’t just somebody who respects the Truth.

All too many people respect the Truth.

They respect the Truth that the U.S. government planted explosives in the World Trade Center, the Truth that the stars control human destiny (ironically, the exact reverse will be true if everything goes right), the Truth that global warming is a lie… and so it goes.

A rationalist is somebody who respects the processes of finding truth. They respect somebody who seems to be showing genuine curiosity, even if that curiosity is about a should-already-be-settled issue like whether the World Trade Center was brought down by explosives, because genuine curiosity is part of a lovable algorithm and respectable process. They respect Stuart Hameroff for trying to test whether neurons have properties conducive to quantum computing, even if this idea seems exceedingly unlikely a priori and was suggested by awful Gödelian arguments about why brains can’t be mechanisms, because Hameroff was trying to test his wacky beliefs experimentally, and humanity would still be living on the savanna if ‘wacky’ beliefs never got tested experimentally.

Or consider the controversy over the way CSICOP (Committee for Skeptical Investigation of Claims of the Paranormal) handled the so-called Mars effect, the controversy which led founder Dennis Rawlins to leave CSICOP. Does the position of the planet Mars in the sky during your hour of birth, actually have an effect on whether you’ll become a famous athlete? I’ll go out on a limb and say no. And if you only respect the Truth, then it doesn’t matter very much whether CSICOP raised the goalposts on the astrologer Gauquelin—i.e., stated a test and then made up new reasons to reject the results after Gauquelin’s result came out positive. The astrological conclusion is almost certainly un-true… and that conclusion was indeed derogated, the Truth upheld.

But a rationalist is disturbed by the claim that there were rational process violations. As a Bayesian, in a case like this you do update to a very small degree in favor of astrology, just not enough to overcome the prior odds; and you update to a larger degree that Gauquelin has inadvertantly uncovered some other phenomenon that might be worth tracking down. One definitely shouldn’t state a test and then ignore the results, or find new reasons the test is invalid, when the results don’t come out your way. That process has bad systematic properties for finding truth—and a rationalist doesn’t just appreciate the beauty of the Truth, but the beauty of the processes and cognitive algorithms that get us there.[1]

The reason why rationalists can have unusually productive and friendly conversations at least when everything goes right, is not that everyone involved has a great and abiding respect for whatever they think is the True or the Optimal in any given moment. Under most everyday conditions, people who argue heatedly aren’t doing so because they know the truth but disrespect it. Rationalist conversations are (potentially) more productive to the degree that everyone respects the process, and is on mostly the same page about what the process should be, thanks to all that explicit study of things like cognitive psychology and probability theory. When Anna tells me, “I’m worried that you don’t seem very curious about this,” there’s this state of mind called ‘curiosity’ that we both agree is important—as a matter of rational process, on a meta-level above the particular issue at hand—and I know as a matter of process that when a respected fellow rationalist tells me that I need to become curious, I should pause and check my curiosity levels and try to increase them.

Is rationality-use necessarily tied to rationality-appreciation? I can imagine a world filled with hordes of rationality-users who were taught in school to use the Art competently, even though only very few people love the Art enough to try to advance it further; and everyone else has no particular love or interest in the Art apart from the practical results it brings. Similarly, I can imagine a competent applied mathematician who only worked at a hedge fund for the money, and had never loved math or programming or optimization in the first place—who’d been in it for the money from day one. I can imagine a competent musician who had no particular love in composition or joy in music, and who only cared for the album sales and groupies. Just because something is imaginable doesn’t make it probable in real life… but then there are many children who learn to play the piano despite having no love for it; “musicians” are those who are unusually good at it, not the adequately-competent.

But for now, in this world where the Art is not yet forcibly impressed on schoolchildren nor yet explicitly rewarded in a standard way on standard career tracks, almost everyone who has any skill at rationality is the sort of person who finds the Art intriguing for its own sake. Which—perhaps unfortunately—explains quite a bit, both about rationalist communities and about the world.

[1] RationalWiki really needs to rename itself to SkepticWiki. They’re very interested in kicking hell out of homeopathy, but not as a group interested in the abstract beauty of questions like “What trials should a strange new hypothesis undergo, which it will notfail if the hypothesis is true?” You can go to them and be like, “You’re criticizing theory X because some people who believe in it are stupid; but many true theories have stupid believers, like how Deepak Chopra claims to be talking about quantum physics; so this is not a useful method in general for discriminating true and false theories” and they’ll be like, “Ha! So what? Who cares? X is crazy!” I think it was actually RationalWiki which first observed that it and Less Wrong ought to swap names.

(Mainstream status here.)

Part of the sequence Highly Advanced Epistemology 101 for Beginners

Next post: “Firewalling the Optimal from the Rational”

Previous post: “Skill: The Map is Not the Territory”

Eliezer Yudkowsky6 Oct 2012 9:59 UTC
97 points
135 comments5 min readLW linkArchive